Methods and systems for selecting an optimized scoring function for use in ranking item listings presented in search results

ABSTRACT

Methods and systems for simulating a search, for the purpose of evaluating one or more scoring functions used in ordering item listings for presentation in a search results page are described. Consistent with some embodiments, a simulation platform includes a real-time simulation module that receives search result sets for search queries that result in the conclusion of a transaction. The result set is then processed by the simulation platform with one or more test scoring functions, such that the resulting position of the item listing that has resulted in the transaction can be compared with the actual position at which the item listing was displayed in the actual search results. For each test scoring function, an average rank shift metric is determined, and displayed, thereby providing a metric with which to base decisions about which scoring functions to use in the production system.

TECHNICAL FIELD

The present disclosure generally relates to data processing techniques.More specifically, the present disclosure relates to methods and systemsfor simulating how search results are processed for the purpose ofselecting an optimized scoring function for use in ordering the searchresults when presenting the search results to an end-user of acomputer-based trading or e-commerce application.

BACKGROUND

Advancements in computer and networking technologies have enabledpersons to conduct commercial and financial transactions “on-line” viacomputer-based applications. This has given rise to a new era ofelectronic commerce (often referred to as e-commerce.) A number ofwell-known retailers have expanded their presence and reach by operatingwebsites that facilitate e-commerce. In addition, many new retailers,which operate exclusively online, have come into existence. The businessmodels utilized by enterprises operating online are almost as varied asthe products and services offered. For instance, some products andservices are offered at fixed prices, while others are offered viavarious auction methods, and still others are offered via a system ofclassified ad listings. Some enterprises specialize in the selling of aspecific type of product (e.g., books) or a specific service (e.g., taxpreparation), while others provide a myriad of categories of items andservices from which to choose. Some enterprises serve only as anintermediary, providing a marketplace to connect sellers and buyers,while others sell directly to consumers.

Despite the many technical advances that have improved the state ofe-commerce, a great number of technical challenges and problems remain.One such problem involves determining how to best present products andservices (e.g., items) that are being offered for sale, so as tomaximize the likelihood that a transaction (e.g., the sale of a productor service) will occur. For instance, when a potential buyer performs asearch for a product or service, it may often be the case that thenumber of item listings that satisfy the potential buyer's query farexceeds the number of item listings that can practically be presented ona search results page. Furthermore, it is well established that thepresentation of an item listing in a search results page—for example,the order or placement of the item listing in a list of listings, thefont, font size, or color of the listing, and so on—can affect whetherpotential buyers select the listing, and ultimately purchase the listedproduct or service.

In order to maximize the number of transactions that are concluded,application developers make significant efforts to determine thecharacteristics (or attributes) of item listings that are the bestindicators of the likelihood that an item listing will result in theconclusion of a transaction. Item listings with the desired itemcharacteristics are then presented in the most prominent positions ofthe search results page (or pages) to maximize the likelihood that thoseitem listings will be selected, and ultimately a transaction will beconcluded. Accordingly, search engines used in conjunction withe-commerce applications may use some sort of parametric scoring functionto assign a score to each item listing that satisfies a particularquery. For example, the desired item characteristics are formulated asparameters in the scoring function such that, those item listings havingthe most desirable item characteristics will be assigned the highestscore based on the scoring function, and therefore will be presentedmost prominently in the search results page or pages.

DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings, in which:

FIG. 1 is a block diagram of a computer-based network environmentincluding a simulation platform for use with a computer-based on-linetrading application, according to an embodiment of the invention;

FIG. 2 illustrates an example of the general format of a scoringfunction for use in assigning a ranking score to an item listing thatsatisfies a search query processed with a computer-based on-line tradingapplication, according to an embodiment of the invention;

FIG. 3 is a block diagram showing an example of the various functionalcomponents or modules that comprise a simulation platform, according toan embodiment of the invention; FIG. 4 is a block diagram showing anexample of a set of search results (e.g., item listings) that satisfy asearch query, both before and after the search result set has beenprocessed with a production scoring function, according to an embodimentof the invention;

FIG. 5 is a block diagram showing an example of a set of search results(e.g., item listings) that satisfy a search query, both before and afterthe search result set has been processed with one of three scoringfunctions, according to an embodiment of the invention;

FIG. 6 is a table showing how an average rank shift metric is derivedfor a predetermined number of search queries, for one or more testfunctions, according to an embodiment of the invention;

FIG. 7 is a flow diagram illustrating the method operations involved ina method for evaluating one or more test scoring functions for use inordering item listings to be presented in search results, according toan embodiment of the invention;

FIG. 8 is a flow diagram illustrating the method operations involved ina method for generating the weighting factors used with a new scoringfunction; and

FIG. 9 is a block diagram of a machine in the form of a computer systemwithin which a set of instructions, for causing the machine to performany one or more of the methodologies discussed herein, may be executed.

DETAILED DESCRIPTION

Methods and systems for simulating how search results are processed forthe purpose of generating and selecting an optimized scoring functionfor use in ordering the search results when presenting the searchresults to an end-user of a computer-based trading or e-commerceapplication are described. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of the various aspects of different embodimentsof the present invention. It will be evident, however, to one skilled inthe art, that the present invention may be practiced without thesespecific details.

Consistent with some embodiments of the invention, an on-line trading ore-commerce application utilizes a parametric scoring function to assigna score to each item listing that satisfies an end-user's search query.The score, which may be referred to as a listing performance score orbest match score, is used to rank and order all of the item listingsthat satisfy a given search query so that the item listings with thebest (e.g., highest, or in some instances, lowest) score can bepresented in the most prominent positions of the search results page orpages. Typically this means that the item listings with the highestscore are presented at the top of a list of item listings presented inthe search results page or pages. However, in various alternativeembodiments, different scoring algorithms may be used such that the itemlistings that are to be displayed in the most prominent positions of thesearch results page or pages will be those item listings assigned thelowest scores. Additionally, in some embodiments, item listings may bepresented in some manner other than a top-down list.

The scoring functions used in ranking and ordering search results arereferred to as parametric scoring functions because the scoringfunctions are expressed with weighted parameters that are selected toreflect a measure of various characteristics (sometimes referred to asitem attributes) of an item listing. The parameters are chosen toreflect item characteristics, and in some cases seller characteristics,that generally correlate with whether or not an item listing is likelyto be selected when presented in the search results, and ultimatelywhether the item listing, if displayed to an end-user in a searchresults page, is likely to result in the conclusion of a transaction(e.g., a conversion or sale). For instance, one parameter used in ascoring function may represent a measure of an item listing's relevancewith respect to the exact search terms provided in the end-user's searchquery. Accordingly, if the item listing is determined to be particularlyrelevant in light of the search terms provided by the end-user, theparameter that represents the measure of relevance will have a positiveimpact on the overall listing performance score of the item listing.Consequently, all else equal, an item listing with a high relevancescore is likely to be positioned more prominently in the search resultsthan an item listing with a low relevance score. Because the scoringfunctions are used in arranging the item listings in the search resultspage or pages, it is important that the scoring functions include theproper parameters, and that the parameters are properly weighted, forexample, to emphasize the item characteristics having the greatestimpact on the overall desirability of an item listing.

Other parameters may be chosen as being representative of an itemlisting's overall quality, for example, based on such item attributes asthe price of an item, the shipping cost of an item, the stated conditionof an item, and/or the length of time that an item listing has beenactive. In yet another example, one or more parameters may represent ameasure of demand for an item listing. The demand may be measured as thenumber of search impressions that a particular item listing has received(e.g., the number of times an item listing has appeared in the searchresults of a search request), the number of times that an end-user hasselected an item listing after the item listing was presented in asearch results page, and/or the number of times that an item listing, ora similar listing, has resulted in the conclusion of a transaction(e.g., the sale of an item or service offered via the item listing).Those skilled in the art will readily appreciate that the exact type andnature of parameters that might be used in a parametric scoring functionfor ranking and ordering search results is virtually unlimited. Somespecific example parameters for use with a scoring function for rankingand ordering search results containing item listings for an on-linetrading or e-commerce application are described in connection with thedescription of FIG. 2 below.

In some embodiments of the invention, the parameters of the scoringfunctions are weighted, for example, by multiplying each parameter by aweighting factor to give greater emphasis to certain parameters inderiving the overall listing performance score assigned to each itemlisting. Additionally, in some embodiments, different scoring functionsare used for different listing types, or listing formats. For instance,in some embodiments, different scoring functions may be used to processitem listings having a fixed-price and item listings offering items atauction with no fixed price. In addition, in some embodiments, itemlistings may be grouped based on categories, such that all item listingsassociated with a certain group of categories are assigned to aparticular group (sometimes referred to as a vertical to reflect aparticular market segment). For instance, one group may be establishedfor clothing, shoes and accessories. Accordingly, any item listing thatis assigned to a category for clothes, a category for shoes, or acategory for accessories, will automatically be part of the clothes,shoes and accessories group. The scoring function used in the processingof item listings in the clothes, shoes and accessories group may bedifferent from the scoring function used to assign a listing performancescore for item listings in other categories or groups, such as acategory or group for jewelry, or automobiles, and so forth.

In some embodiments, a scoring profile defines a set of scoringfunctions and maps the scoring functions to the listing types, groups orcategories of item listings with which the scoring functions are to beused. For example, a scoring profile may define a first scoring functionfor a clothes, shoes and accessories group, a second scoring functionfor an automobiles group, and a third scoring function for a jewelrygroup. In various embodiments, the mapping of scoring functions tolisting types, categories or groups, may be defined at a more granularlevel. For example, in some embodiments, scoring functions may beindividually defined and mapped to item listings in certainsub-categories, and so on.

Consistent with some embodiments of the invention, when the itemlistings are presented in the search results pages, an activity monitordetects certain end-user activities and generates event data indicatingwhat type of activity was detected and to which item listing theactivity was directed or with which item listing the activity wasassociated. For example, the activity monitor may detect when anend-user selects (e.g., with a cursor control device, such as a mouse ortouch screen) a particular item listing presented in the search resultspage or pages. Such an event is often referred to as a “view item page”event. Similarly, the activity monitor may detect and keep track ofthose item listings that, after being displayed in a search results pageas a result of processing a particular search query, result in theconclusion of a transaction. For instance, after an end-user hasselected an item listing to view, the end-user may bid on an itemoffered via auction, or select a fixed-priced item that is offered via abuy-it-now feature. An event that results in the sale of an item isgenerally referred to as a conversion event. An item listing thatresults in the conversion of an item—that is, the sale of an item orservice offered via the item listing—is referred to herein as aconverting item listing.

When a conversion event occurs, the activity monitor detects and storesthe particular position of the item listing that ultimately resulted inthe conclusion of a transaction. For example, the position of the itemlisting may be the listing slot number identifying the listing slot atwhich the item listing was displayed in the search results page. If, forexample, the item listing was the very first item listing in the searchresults page, the listing slot number for the item listing would be one.In some embodiments, listing slots may be assigned by page and number,such that each item listing is assigned to a particular page of searchresults, and a particular listing slot (e.g., position or order in alist) on the assigned page. For instance, in such a scenario, the itemlisting assigned the highest listing performance score by the productionscoring function would be assigned listing slot one, on search resultspage number one. In any case, when a conversion event occurs, theposition of the item listing that resulted in the conversion isdetected, and passed to the real-time simulation platform with thesearch result set (e.g., the item listings that satisfied the end-user'squery).

When the simulation platform receives the search result set, a datatracking module first identifies and retrieves all relevant dataassociated with the item listings included in the search result set. Insome embodiments, the relevant data is the data required as input to theone or more test scoring functions being evaluated by the simulationplatform. In any case, once the data is identified and retrieved, theitem listings (and associated item attribute data) are processed withone or more test scoring functions. As a result, each item listing inthe set of search results is assigned a position at which the itemlisting would appear if the item listing was to be presented in a searchresults page. The position (e.g., listing slot) assigned by the testscoring function of the particular item listing that resulted in theconclusion of a transaction can then be compared to the originalposition that was assigned to the converting item listing by theproduction scoring function. If, for example, the test scoring functionassigns a lower listing slot number—indicating that the item listingwould appear closer to the top of the list of search results, then thetest scoring function has outperformed the production scoring functionfor that one particular query. By simulating the processing of searchresults for a sufficiently large number of queries, the performance ofone or more test scoring functions can be compared with the performanceof the production scoring function. If the results indicate that one ofthe test scoring functions consistently outperforms the productionscoring function, for example, by positioning the converting itemlisting in a lower listing slot number, then a decision may be made tomove the test scoring function to the production system.

In yet another aspect of the invention, an offline simulator receivessets of search results and the corresponding converting item listingsfor those search results, and generates weighted parameters for newscoring functions based on some pre-defined constraints. For instance,the sets of search results are processed by the offline simulator moduleto derive new weighting factors for a given set of parameters used in ascoring function. The parameters may be the same parameters as those inuse by a production scoring function, or the parameters may be differentfrom those in use by the production scoring function. The set ofconstraints, for example, might be defined to derive the weightingfactors that result in the largest positive average rank shift for a setof search results. For instance, the weighting factors may be selectedsuch that, on average, the converting item listing appears in thehighest position on the list (corresponding with the lowest listing slotnumber). Of course, other constraints might be specified to achieveother business objectives. Various other inventive aspects of themethods and systems described herein will become readily apparent fromthe description of the figures that follows.

FIG. 1 is a block diagram of a computer-based network environment 10including a simulation platform 12 for use with a computer-based on-linetrading application 14, according to an embodiment of the invention. Asillustrated in FIG. 1, the on-line trading application 14 is shown toreside on a server system 16 that is coupled by means of a network 18 toa client computer system 20. Accordingly, in some embodiments, theon-line trading application is a web-based application that isaccessible via a conventional web browser application 22, for example,residing and executing at client computer system 20. Although shown inFIG. 1 as residing on a single server system 16, in various embodimentsthe on-line trading application 14 may be implemented in a clusterconfiguration such that the trading application is distributed across anumber of server systems. As shown in FIG. 1, the on-line tradingapplication 14 is coupled to a simulation platform 12 residing andexecuting on a separate server system. In some embodiments, thesimulation platform 12 will reside and execute at one or more separateprocessor-based computer systems, as shown in FIG. 1. However, in somealternative embodiments, one or more of the various modules comprisingthe simulation platform 12 may reside and execute on the same serversystem 16 as the on-line trading application 14.

In this example, the online trading application 14 comprises two primarymodules—an on-line trading engine module 24, and an item listingpresentation management module 26. In some embodiments, the on-linetrading engine module 24 may consist of a variety of sub-components ormodules, which provide some of the many functions included as part ofthe on-line trading application 14. As described more completely below,each module may comprise software instructions, computer hardwarecomponents, or a combination of both. To avoid obscuring the inventionin unnecessary detail, only a few of the on-line trading enginefunctions (germane to the invention) are described herein. For example,the on-line trading engine module 24 may include an item listingmanagement module (not shown in FIG. 1) that facilitates the receivingand storing of data representing item attributes, which collectivelyform an item listing. When a user desires to list a single item, ormultiple items, for sale, the user will provide information about theitem(s) (e.g., item attributes). Such information may be submitted viaone or more forms of one or more web pages, or via drop down lists, orsimilar user interface elements. The item listing management modulereceives the item attributes and stores the item attributes togetherwithin a database 28 as an item listing. In some instances, the itemlistings may be stored in an item listing database table 30. Asdescribed in greater detail below, the item attributes of each itemlisting are analyzed to determine a ranking score assigned to itemlistings and used in determining the position of item listings (e.g.,the order of item listings) when the item listings are being presentedin a search results page. In addition, the item attributes of each itemlisting may be utilized by the simulation platform 12 to evaluate theperformance of one or more alternative scoring functions, or testscoring functions.

The on-line trading engine module 24 may also include one or moremodules for detecting, receiving and storing activity data that is usedas an input to a scoring function to measure the likelihood that an itemlisting will, if presented in a search results page, result in atransaction being concluded. For instance, in some embodiments, dataassociated with user-initiated activities are analyzed and captured forthe purpose of predicting future user activities. If a user submits asearch request including certain search terms, and then proceeds toconclude a transaction for a particular item (e.g., purchase the item),information from the user's interaction with the online tradingapplication 14 will be captured and stored for the purpose of predictingfuture actions by other users. Some of the data used in this capacity isgenerally referred to as relevance data 32 because it is used todetermine a measure of relevance between search terms used in a searchquery, and individual item listings. For instance, if a potential buyersubmits a search request with the search terms, “mobile phone”, itemlistings that have certain item attributes are more likely to result inthe conclusion of a transaction if presented in a search results page inresponse to the search request. For instance, continuing with theexample search terms, “mobile phone”, given the specific search termsused in the search query, item listings that have been designated asbeing in a certain category of items, such as “Electronics”, or evenmore specifically, “Mobile Phones”, are more likely to result in atransaction if presented in a search results page than item listings inother categories, for example, such as “Automobiles” or “Jewelry”.Similarly, given the search terms, “mobile phone”, item listings withtitles that include the search terms may prove more likely to result ina transaction than item listings without the search terms in the title.Accordingly, in some embodiments, the on-line trading engine module 24includes one or more modules for receiving and analyzing historicalactivity data to generate what is referred to herein as relevance data32. The relevance data 32 may be used as a parameter in a scoringfunction, or as an input to a parameter, to derive a measure of thelikelihood that item listings with certain item attributes will resultin a transaction if displayed in response to certain search terms beingsubmitted in a search request.

The on-line trading engine module 24 may also include one or moremodules for receiving and storing data representing, among other things,a measure of a seller's performance of obligations associated withtransactions in which the seller has previously participated. Forinstance, in some embodiments, when a transaction is concluded, a buyermay be prompted to provide feedback information concerning theperformance of a seller. The buyer may, for example, rate the accuracyof the seller's description of an item provided in the item listing. Forinstance, if the item received by the buyer is in poor condition, butwas described in the item listing as being in good condition, the buyermay provide feedback information to reflect that the seller'sdescription of the item in the item listing was inaccurate. Thisinformation may be used as a parameter to a scoring function used toassign an item listing performance score to item listings whenprocessing a search query. For instance, in some cases, the sellerfeedback information may be used to determine a listing performancescore for another item listing of the same seller. Such information maybe stored in a database 28, as indicated in FIG. 1 by the seller qualitydata with reference number 34.

As illustrated in FIG. 1, the database is also shown to include businessrules data 36. In some embodiments, the business rules data 36 ismanaged and used by a business rules management module (not shown) forthe purpose of promoting and/or demoting item listings that satisfy asearch query. For instance, when determining the order or arrangement ofitem listings for presentation on a search results page, an item listingmay be promoted—presented in a more prominent position—or,demoted—presented in a less prominent position—based on the evaluationof a business rule that is dependent upon certain business rule data 36.In some embodiments, item attributes and seller attributes may be usedin conjunction with the business rules data, for the purpose ofevaluating business rules. In some embodiments, the promotion ordemotion may be effected by utilizing a business rules score as aparameter in a scoring function, such that the business rules score ismultiplied by one or more other parameters of the scoring function.Business rules may be used to promote certain business policies and toimpact user's behavior. For instance, a business rule that provides apromotion to item listings that are offering free shipping will likelyhave the effect of persuading sellers to offer free shipping to havetheir item listings appear in the most prominent positions of the searchresults page. Similarly, demoting item listings based on negative sellerfeedback information will typically motivate sellers to perform theirobligations as agreed upon.

Referring again to FIG. 1, the second primary module of the on-linetrading application 14 is an item listing presentation management module26. The item listing presentation management module 26 includes logicused to process search queries and assign a listing performance score(sometimes referred to as a best match score) to item listings thatsatisfy a search query, and to use the listing performance scoreassigned to each item listing that satisfies the search query todetermine the order of item listings when the item listings arepresented in a search results page. Accordingly, the item listingpresentation management module 26 includes a scoring profile thatdefines one or more production scoring functions used to process the setof item listings that satisfy a search query. The item listingpresentation management module 26 utilizes the production scoringfunction(s) to analyze the various attributes of the item listings andassign each item listing a listing performance score that ultimatelydetermines the listing's position in the search results page or pages.

For instance, in some embodiments, a user operates a web browserapplication 22 on a client system 20 to interact with the on-linetrading application 24 residing and executing on the server system 16.As illustrated by the example user interface with reference number 38, auser may be presented with a search interface 38, with which the usercan specify one or more search terms to be used in a search requestsubmitted to the on-line trading application 14. In some embodiments, inaddition to specifying search terms, users may be able to select certainitem attributes, such as the desired color of an item, the itemcategories that are to be searched, and so on. After receiving andprocessing the search request, the on-line trading application 14communicates a response to the web browser application 22 on the clientsystem 20. For instance, the response is an Internet document or webpage that, when rendered by the browser application 22, displays asearch results page 40 showing several item listings that satisfy theuser's search request. As illustrated in the example search results page40 of FIG. 1, the item listings are arranged or positioned on the searchresults page in an order determined by the item listing presentationmanagement module 26. In particular, the order of the item listings aspresented on the search results page 40 is determined based on theproduction scoring functions 27 specified as part of the scoring profile25. The item listings are, in some embodiments, presented by apresentation module, which may be, or operate in conjunction with, a webserver or an application server. 10023] In general, the item listingsare presented in the search results page in an order based on a listingperformance score (e.g., a ranking score) that is assigned to each itemlisting that satisfies the query—also referred to as the result set of asearch query, or, search results set. In some embodiments, the itemlistings will be arranged in a simple list, with the item listing havingthe highest listing performance score (e.g., ranking score) appearing inthe first listing slot, for example, at the top of the list, followed bythe item listing with the next highest listing performance score, and soon. In some embodiments, several search results pages may be required topresent all item listings that satisfy the query. Accordingly, only asubset of the set of item listings that satisfy the query may bepresented in the first page of the search results pages. In someembodiments, the item listings may be ordered or arranged in some othermanner, based on their listing performance scores. For instance, insteadof using a simple list, in some embodiments the item listings may bepresented one item listing per page, or, arranged in some manner otherthan a top-down list.

The listing performance score assigned to each item listing may be basedon several component scores (e.g., parameters) including, but by nomeans limited to: a relevance score, representing a measure of therelevance of an item listing with respect to search terms provided inthe search request; a listing quality score, representing a measure ofthe likelihood that an item listing will result in a transaction basedat least in part on historical data associated with similar itemlistings; and, a business rules score, representing a promotion ordemotion factor determined based on the evaluation of one or morebusiness rules. As used herein, a component score is a score that isused in deriving the overall ranking score for an item listing. However,a component score in one embodiment may be a ranking score in anotherembodiment. Moreover, it will be appreciated that the parametersspecified for any given scoring function may be defined at variouslevels of granularity, such that a parameter may represent a componentscore, or a sub-component, and so on. For instance, in some embodiments,the ranking score may be equivalent to a single component score, such asthe listing quality score. Alternatively, various components thatcomprise the listing quality score may be parameters with their ownweighting factors.

FIG. 2 illustrates an example of the general format of a scoringfunction for use in assigning a ranking score, or listing performancescore, to an item listing that satisfies a search query processed with acomputer-based on-line trading application, according to an embodimentof the invention. As illustrated in FIG. 2, the example format for ascoring function is a formula or equation having any number of weightedparameters. In one particular embodiment, an example scoring function 44may involve a weighted relevance score or parameter 46, a weightedlisting quality score or parameter 48, and a weighted business rulesscore or parameter 50. The example scoring function 44 of FIG. 2 may beconsistent with the description of the formula for deriving what isreferred to as a best match score in related U.S. patent applicationSer. No. 12/476,046, filed on Jun. 1, 2009, entitled, “METHODS ANDSYSTEMS FOR DERIVING A SCORE WITH WHICH ITEM LISTINGS ARE ORDERED WHENPRESENTED IN SEARCH RESULTS”, which is hereby incorporated herein byreference. It will be readily appreciated by those skilled in the artthat the example scoring function of FIG. 2 is but one of many scoringfunctions, which may have any number and type of parameters for use inassigning a ranking score to an item listing for use in ordering theitem listings when presenting them in search results.

FIG. 3 is a block diagram showing an example of the various functionalcomponents or modules that comprise a simulation platform 12 for usewith an on-line trading or e-commerce application, according to anembodiment of the invention. As illustrated in FIG. 3, the simulationplatform 12 includes a data management module 60, a real-time scoringfunction evaluation module 62 and an off-line simulator module 64. Insome embodiments of the invention, when an end-user specifies one ormore keywords for a search, or selects various desired item attributes,a search query is executed by a search engine (not shown) of the itemlisting presentation management module 26. The item listing presentationmanagement module 26 first identifies the item listings that satisfy thesearch query. Such item listings are referred to herein as the searchresult set. Once the search result set is identified, the productionscoring function 27 is used to assign to each item listing in the searchresult set a listing performance score, or ranking score. Finally, asearch results page is generated showing the item listings that satisfythe search query in an order that is based on their assigned listingperformance score generated using the production scoring function 27.

As indicated by the arrow with reference number 61, when a particularitem listing in the search results pages is selected and ultimatelyresults in the conclusion of a transaction, the search result set ispassed to the simulation platform module 12. In particular the dataforwarded to the simulation platform module 12 may include the searchquery, the search result set (e.g., the item listings that satisfy thesearch query), and the position of the item listing that resulted in theconclusion of the transaction. In some instances, other activity datamay be communicated to the simulation platform. For example, a view itemevent occurs when a user selects a particular item listing to view, butdoes not finalize a transaction. In some instances this information mayalso be passed to the simulation platform. When the query data isreceived at the data management module 60 of the simulation platformmodule 12, the real time data processing module 66 processes the data byretrieving any relevant data from database 22 that may be used as inputdata to the one or more test scoring functions being evaluated at thesimulation platform module 12.

The real-time scoring function evaluation module 62 processes thereceived data with the current production scoring function 68 and one ormore test scoring functions, such as the test scoring functionsillustrated in FIG. 3 with reference numbers 70, 72 and 74. Theevaluation module 62 calculates a rank shift for each test scoringfunction. The rank shift is a metric by which this test scoring functioncan be compared to the current production scoring function 68. Forexample, the position of the item listing that resulted in theconclusion of a transaction as determined by the current productionscoring function 68 is taken as the baseline. Accordingly, the positionof the particular item listing that resulted in the conclusion of atransaction as determined by one or more of the test scoring functionscan be compared with the position of the same item listing determined bythe current production scoring function 68. Assuming one of the testscoring functions results in positioning the particular item listingthat resulted in the conclusion of a transaction in a more prominentlisting slot (e.g., a lower numbered listing slot, based on a higherlisting performance score), then that test scoring function would haveoutperformed the production scoring function for the particular query.By evaluating one or more test scoring functions over a large number ofqueries an average rank shift metric for each test scoring function canbe derived. The average rank shift metric represents the average numberof positions that a converting item listing is shifted from its baselineposition (determined with the production scoring function) as a resultof processing the queries with the test scoring function. A furtherexample is presented in connection with the description of FIGS. 4 and5.

In addition to the scoring function evaluation module 62, the simulationplatform includes an off-line simulator for generating new test scoringfunctions. Specifically, a data importing and enhancing module 65 of theoff-line simulator 64 requests data from the off-line data exportingmodule 63 of the data management module 60, and imports datarepresenting sets of search results and corresponding converting itemlistings. The imported data is then processed with a parameteroptimization module 67 to generate a new test scoring function 69. Theparameter optimization module 67 processes sets of search results inaccordance with one or more defined constraints, to derive weightingfactors for parameters of a new scoring function that meet the definedconstraints. For instance, in some cases, the constraints may be definedto derive the weighting factors for use in weighting certain parametersof a new scoring function, such that the new scoring function has thehighest average rank shift metric compared to the baseline representedby the production scoring function. Once the weighting factors arederived, the new test scoring function can be evaluated with real-timedata against other test scoring functions and the production scoringfunction.

FIG. 4 is a block diagram showing an example of a set of search results90 (e.g., item listings) that satisfy a search query 91, both before andafter the search result set has been processed with a production scoringfunction, according to an embodiment of the invention. As illustrated inFIG. 4, an end-user's search query 91 is processed to generate anunranked result set 90. For example, the unranked result set 90 includesthe set of item listings that satisfy the search query 91 shown in noparticular order. After the item listings that satisfy the search queryhave been identified, a production scoring function 92 is utilized toassign to each item listing in the unranked result set a listingperformance score. Based on the listing performance score assigned toeach item listing, the item listings are ordered for presentation in thesearch results page. For example, in FIG. 4, the ranked search resultset is shown with reference number 94. For instance, item listing threeis shown to be positioned in listing slot one, representing the topposition in the search results list.

As indicated in FIG. 4, item listing 5 in listing slot number four isselected by an end-user and ultimately results in the conclusion of atransaction. For instance, in this example, the user has selected itemlisting five and has purchased the item associated with item listingfive. As such, item listing five, which is in listing slot number four,is referred to as a converting item listing. When this occurs, thesearch query 91, the ordered search result set 94 and the position, andcorresponding listing performance score, for the converting item listing(e.g., item listing five in the example) are communicated to thesimulation platform, thereby enabling the real-time evaluation of one ormore test scoring functions.

FIG. 5 is a block diagram showing an example of a set of search results(e.g., item listings) that satisfy a search query, both before and afterthe search result set has been processed with one of three scoringfunctions by a simulation platform, according to an embodiment of theinvention. As illustrated in FIG. 5, the unranked result set 100 isprocessed at the simulation platform module using the production scoringfunction 102 as well as two test scoring functions 104 and 106. Thevarious search results show how the individual scoring functions (e.g.,102, 104 and 106) order the item listings. For example, the productionscoring function 102 generates the exact same result that was realizedby the production scoring function in use by the online tradingapplication (as illustrated in FIG. 4). Specifically, item listing five,which is the item listing that resulted in the conclusion of atransaction, appears in listing slot number four. As such, theproduction scoring function 102 is taken as a baseline to which the testscoring functions can be compared. More specifically, for thisparticular search query and result set, the baseline listing slot islisting slot number four, which is the position of the item listing inthe actual search results page that resulted in a transaction beingconcluded.

When the item listings are ordered with test scoring function one, withreference number 104, item listing five is positioned in listing slotnumber two—an improvement over the baseline position. As such, for thisexample search query, the rank shift metric for test scoring functionone is a positive two, reflecting the fact that item listing five hasmoved from listing slot number four to listing slot number two when theitem listings were processed with test scoring function number one(e.g., reference number 104). As item listing five represents the itemlisting selected by the end-user, its placement in listing slot numbertwo represents an improvement, as listing slot number two is deemed tobe a more prominent position than listing slot number four. As such, forthis particular query, test scoring function number one has outperformedthe production scoring function.

As illustrated in FIG. 5, when the item listings are processed with testscoring function number two, with reference number 106, item listingfive is positioned in listing slot number five. Accordingly, for testscoring function number two, the rank shift metric for this particularquery is negative one, reflecting the change of the position of itemlisting five. In this particular example, for this particular query, theproduction scoring function 102 has outperformed test scoring functiontwo, with reference number 106.

While FIG. 5 illustrates an example of processing a single query andassociated search result set, it will be appreciated that the describedprocess is repeated for a predetermined number of search queries toestablish an average rank shift metric for each test scoring functionthat is being tested. For instance, FIG. 6 is a table showing how anaverage rank shift metric is derived for a predetermined number ofqueries, for one or more test functions, according to an embodiment ofthe invention. As illustrated in the table of FIG. 6, the simulation andevaluation of the test scoring functions is repeated for a predeterminednumber of queries. Accordingly, for each test scoring function and foreach query a rank shift is observed. After enough queries have beenprocessed, an average rank shift for each test scoring function can becomputed. As illustrated in FIG. 6, the average rank shift for testscoring function one, for the “N” queries, is a positive 1.2, reflectinga slight improvement over the baseline of the production scoringfunction. Similarly, test scoring function number two has exhibited anaverage rank shift of negative 2.3, reflecting a slightly poorerperformance than the baseline of the production scoring function.Finally, test scoring function N has exhibited a −1.1 average rankshift, representing a slightly poorer performance than the productionscoring function.

It will be appreciated by those skilled in the art, that the averagerank shift can be used to evaluate the various test scoring functionsand determine the test scoring function that positions the convertingitem listings in the most prominent listing slots. As such, if a testscoring function exhibits a positive average rank shift over astatistically significant sample of search queries, the test scoringfunction may be put into the production system, replacing the productionscoring function that it outperforms.

FIG. 7 is a flow diagram illustrating the method operations involved ina method for evaluating one or more test scoring functions for use inordering item listings to be presented in search results, according toan embodiment of the invention. The method begins at operation 120 whenthe real-time simulator (e.g., simulation platform 12 in FIG. 3)receives a search result set for a search query, along with anindication of a position (e.g., a listing slot number) for an itemlisting that has resulted in the conclusion of a transaction. Forexample, if a user performs a search with the keyword, “iPod 8 GB,” andthen selects the item listing in listing slot number five, and concludesa transaction, the item listing for listing slot number five is theconverting item listing.

At method operation 122, the simulation platform module 12 processes thesearch result set with one or more test scoring functions and thecurrent production scoring function. In some embodiments, the simulationoperation is not performed with the production scoring function becausethe position of each converting item listing is already known and passedto the simulation module. However, in other embodiments, the productionscoring functions are used to process the search result set at thesimulation platform to establish the baseline. In any case, the itemlistings are ordered using each of the scoring functions. At methodoperation 124, for each test scoring function, a rink shift is derivedby determining the difference in the position of the converting itemlisting as determined by the production scoring function and theposition of the converting item listing as determined by a test scoringfunction. For example, if the converting item listing appeared inlisting slot number three with the test scoring function, and listingslot number five with the production scoring function, the rank shiftwould be positive 2 (+2), reflecting the change from listing slot fiveto listing slot three.

At method operation 126, the simulation module determines whether apredetermined number of queries have been processed. For instance, thepredetermined number of queries may be set to reflect some statisticallysignificant sample size to ensure that any resulting average rank shiftmetric for a test scoring function is representative of the scoringfunction's actual performance. If the threshold number of queries hasbeen reached, then at operation 128, for each test scoring function anaverage rank shift is calculated as the sum of the rank shifts for theindividual queries over the predetermined number of queries. Finally atmethod operation 130, the average rank shift for each test scoringfunction is displayed. This allows an administrator, or businessdecision maker, to compare the average rank shift metrics for thevarious test scoring functions with the current production scoringfunction, and decide whether or not to replace the current productionscoring function with one of the test scoring functions.

FIG. 8 is a flow diagram illustrating the method operations involved ina method for generating the weighting factors used with a new scoringfunction. As illustrated in FIG. 8, at method operation 132, an offlinesimulator module receives or imports several sets of search results withcorresponding converting item listings. For instance, because theoffline simulator does not operate in real-time, large quantities ofdata (e.g., search result set) can be imported and processed. Incontrast to the real-time simulator, which evaluates the various scoringfunctions by comparing the performance of the test scoring functions tothe production scoring function or functions, the offline simulator isdesigned to derive one or more sets of weighting factors for use inweighting the parameters of a particular scoring function.

Accordingly, at method operation 134, the imported or received sets ofsearch results are processed in accordance with one or more pre-definedconstraints to derive an optimized set of weighting factors for thespecified parameters of a new scoring function. For instance, theconstraint may be defined to derive the weighting factors that providethe greatest average rank shift metric for a particular set of searchresults, for a particular set of parameters. In this manner, new testscoring functions can be generated to achieve certain businessobjectives that correspond with the defined constraints. At methodoperation 136, the new scoring function is exported to a real-timesimulator (e.g., the scoring function evaluation model 62 in FIG. 3) sothat the new scoring function can be tested in real-time, with real-timeproduction data, against the production scoring function.

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the Internet) and via one or more appropriate interfaces(e.g., Application Program Interfaces (APIs).)

FIG. 9 is a block diagram of a machine in the form of a mobile devicewithin which a set of instructions, for causing the machine to performany one or more of the methodologies discussed herein, may be executed.In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in server-client network environments, or as a peermachine in peer-to-peer (or distributed) network environments. Themachine may be a personal computer (PC), a tablet PC, a set-top box(STB), a Personal Digital Assistant (PDA), a mobile telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The example computer system 1500 includes a processor 1502 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 1501 and a static memory 1506, which communicatewith each other via a bus 1508. The computer system 1500 may furtherinclude a display unit 1510, an alphanumeric input device 1517 (e.g., akeyboard), and a user interface (UI) navigation device 1511 (e.g., amouse). In one embodiment, the display, input device and cursor controldevice are a touch screen display. The computer system 1500 mayadditionally include a storage device (e.g., drive unit 1516), a signalgeneration device 1518 (e.g., a speaker), a network interface device1520, and one or more sensors 1521, such as a global positioning systemsensor, compass, accelerometer, or other sensor.

The drive unit 1516 includes a machine-readable medium 1522 on which isstored one or more sets of instructions and data structures (e.g.,software 1523) embodying or utilized by any one or more of themethodologies or functions described herein. The software 1523 may alsoreside, completely or at least partially, within the main memory 1501and/or within the processor 1502 during execution thereof by thecomputer system 1500, the main memory 1501 and the processor 1502 alsoconstituting machine-readable media.

While the machine-readable medium 1522 is illustrated in an exampleembodiment to be a single medium, the term “machine-readable medium” mayinclude a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more instructions. The term “machine-readable medium” shallalso be taken to include any tangible medium that is capable of storing,encoding or carrying instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresent invention, or that is capable of storing, encoding or carryingdata structures utilized by or associated with such instructions. Theterm “machine-readable medium” shall accordingly be taken to include,but not be limited to, solid-state memories, and optical and magneticmedia. Specific examples of machine-readable media include non-volatilememory, including by way of example semiconductor memory devices, e.g.,EPROM, EEPROM, and flash memory devices; magnetic disks such as internalhard disks and removable disks; magneto-optical disks; and CD-ROM andDVD-ROM disks.

The software 1523 may further be transmitted or received over acommunications network 1526 using a transmission medium via the networkinterface device 1520 utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (“LAN”), a wide area network (“WAN”), theInternet, mobile telephone networks, Plain Old Telephone (POTS)networks, and wireless data networks (e.g., Wi-Fi® and WiMax® networks).The term “transmission medium” shall be taken to include any intangiblemedium that is capable of storing, encoding or carrying instructions forexecution by the machine, and includes digital or analog communicationssignals or other intangible medium to facilitate communication of suchsoftware.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof, show by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

1. A computer-implemented method comprising: receiving a set of itemlistings satisfying a search query and a slot listing number identifyingthe position of a particular item listing that has resulted in aconversion of an item, the set of item listings having been presented ina first search results page with each item listing positioned in anorder determined by processing the set of item listings with aproduction scoring function; processing the set of item listings with atest scoring function to assign to each item listing in the set of itemlistings a position at which each item listing would appear if presentedin a second search results page; based on the position at which theconverting item listing would appear if presented in the second searchresults page, determining a number and direction of positions that theparticular item listing has shifted from its position in the firstsearch results page as a result of the set of item listings beingprocessed with the test scoring function; with the number and directionof positions that the particular item listing has shifted, deriving anaverage rank shift for the test scoring function, the average rank shiftrepresenting an average number and direction of positions that aconverting item listing is shifted as a result of processing apredetermined number of sets of item listings with the test scoringfunction; and displaying the average rank shift.
 2. Thecomputer-implemented method of claim 1, wherein the receiving of theitem listings and the processing of the item listings with the testscoring function occur in substantially real time using input data forthe test scoring function that is equivalent to the input data used bythe production scoring function.
 3. The computer-implemented method ofclaim 1, wherein receiving a set of item listings satisfying a searchquery includes receiving item attribute data associated with each itemlisting for use as input data to the test scoring function.
 4. Thecomputer-implemented method of claim 1, wherein processing the set ofitem listings with the test scoring function to assign to each itemlisting in the set of item listings a position at which each itemlisting would appear if presented in a second search results pageincludes assigning to each item listing in the set of item listings alisting slot number identifying the position in a search results pagewhere an item listing listing would appear if presented in the secondsearch results page.
 5. The computer-implemented method of claim 1,wherein a positive average rank shift for a test scoring functionindicates the test scoring function performed better than the productionscoring function for the predetermined number of sets of item listingsprocessed with the test scoring function.
 6. The computer-implementedmethod of claim 1, wherein a negative average rank shift for a testscoring function indicates the test scoring function performed worsethan the production scoring function for the predetermined number ofsets of item listings processed with the test scoring function.
 7. Thecomputer-implemented method of claim 1, wherein the particular itemlisting that has resulted in a conversion of an item is an item listingthat has been displayed as a result of having been selected from the setof item listings presented in the first search results page.
 8. Thecomputer-implemented method of claim 1, wherein the production scoringfunction and the test scoring function are implemented as equationsinvolving the product of one or more weighted parameters.
 9. Thecomputer-implemented method of claim 8, wherein the one or more weightedparameters are selected from the group consisting of: specified price ofitem, shipping cost, search impression count of an item listing, numberof item views for an item listing, number of previous transactionsconcluded for an item listing, median selling price of item listingsdetermined to be similar to an item listing, and median shipping cost ofitem listings determined to be similar to an item listing.
 10. A dataprocessing system comprising: at least one processor; and amachine-readable medium in communication with the at least oneprocessor, the machine readable medium storing a simulation platformmodule that is executable by the at least one processor, thenetwork-attached storage management module being executed by the atleast one processor to cause operations to be performed, the operationscomprising: receiving a set of item listings satisfying a search queryand a position of a converting item listing that has resulted in aconversion of an item, the set of item listings having been presented ina first search results page with each item listing positioned in anorder determined by processing the set of item listings with aproduction scoring function; processing the set of item listings with atest scoring function to assign to each item listing in the set of itemlistings a position at which each item listing would appear if presentedin a second search results page; based on the position at which theconverting item listing would appear if presented in the second searchresults page, determining a number and direction of positions that theconverting item listing has shifted from its position in the firstsearch results page as a result of being processed with the test scoringfunction; with the number and direction of positions that the particularitem listing has shifted, deriving an average rank shift for the testscoring function, the average rank shift representing an average numberand direction of positions that a converting item listing is shifted asa result of processing a predetermined number of sets of item listingswith the test scoring function; and displaying the average rank shift.11. The data processing system of claim 10, wherein the receiving of theitem listings and the processing of the item listings with the testscoring function occur in substantially real time using input data forthe test scoring function that is equivalent to the input data used bythe production scoring function.
 12. The data processing system of claim10, wherein receiving a set of item listings satisfying a search queryincludes receiving item attribute data associated with each item listingfor use as input data to the test scoring function.
 13. The dataprocessing system of claim 10, wherein processing the set of itemlistings with a test scoring function to assign to each item listing inthe set of item listings a position at which each item listing wouldappear if presented in a second search results page includes assigningto each item listing in the set of item listings a listing slot numberidentifying a listing slot in a search results page where an itemlisting listing would appear if presented in the second search resultspage.
 14. The data processing system of claim 10, wherein a positiveaverage rank shift for a test scoring function indicates the testscoring function performed better than the production scoring functionfor the predetermined number of sets of item listings processed with thetest scoring function.
 15. The data processing system of claim 10,wherein a negative average rank shift for a test scoring functionindicates the test scoring function performed worse than the productionscoring function for the predetermined number of sets of item listingsprocessed with the test scoring function.
 16. The data processing systemof claim 10, wherein the converting item listing that has resulted in aconversion of an item is an item listing that has been displayed as aresult of having been selected from the set of item listings presentedin the first search results page.
 17. The data processing system ofclaim 10, wherein the production scoring function and the test scoringfunction are specified as equations involving the product of one or moreweighted parameters.